Method and system for detection of user-initiated events utilizing automatic content recognition

ABSTRACT

Systems, methods and/or techniques for detection of user-initiated events in relation to a media device are described, for example utilizing a content recognition system. A method may be executed in a media device that is operable to play video content. The method may include receiving user interaction with the media device and the user interaction causes initiation of a feature that affects video content that plays on the media device. The method may include receiving one or more interactive events from a content recognition system in response to the user interaction. The method may comprise executing the one or more interactive events to affect video content that is playing on the media device, and display one or more overlay windows corresponding to the affected video content. One or more video fingerprints that correspond to the affected video content may be communicated to the content recognition system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application makes reference to, claims priority to, and claims thebenefit of U.S. provisional application 61/596,012, filed on Feb. 7,2012.

This application also makes reference to:

-   U.S. patent application Ser. No. 13/730,359, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,422, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,459, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,495, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,530, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,754, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,559, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,579, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,593, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,759, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,627, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,644, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,656, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,670, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,691, filed Dec. 28, 2012;-   U.S. patent application Ser. No. 13/730,702, filed Dec. 28, 2012;    and-   U.S. patent application Ser. No. 13/730,718, filed Dec. 28, 2012.

Each of the above referenced applications is hereby incorporated hereinby reference in its entirety.

FIELD OF THE INVENTION

Certain embodiments of the invention relate to digital audio/videocontent recognition. More specifically, certain embodiments of theinvention relate to a method and system for detection of user-initiatedevents utilizing automatic content recognition.

BACKGROUND OF THE INVENTION

Smart or connected televisions (TVs) may receive data from data networksthat allow a viewer to access broadcast digital content and also receivemultimedia content. Smart TVs may also be capable of receiving andcommunicating information over the internet.

Further limitations and disadvantages of conventional and traditionalapproaches will become apparent to one of skill in the art, throughcomparison of such systems with some aspects of the present invention asset forth in the remainder of the present application with reference tothe drawings.

BRIEF SUMMARY OF THE INVENTION

A system and/or method is provided for detection of user-initiatedevents utilizing automatic content recognition, substantially as shownin and/or described in connection with at least one of the figures, asset forth more completely in the claims.

These and other advantages, aspects and novel features of the presentinvention, as well as details of an illustrated embodiment thereof, willbe more fully understood from the following description and drawings.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a high-level block diagram illustrating an exemplary systemfor detection of user-initiated events, in accordance with one or moreembodiments of the present invention.

FIG. 2 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention.

FIG. 3 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention.

FIG. 4 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention.

FIG. 5 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention.

FIG. 6 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention.

FIG. 7 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention.

FIG. 8 is a flow diagram illustrating exemplary steps in the operationof an exemplary system for detection of user-initiated events, inaccordance with one or more embodiments of the present invention.

FIG. 9 is a block diagram illustrating an exemplary data processingsystem that may be used in accordance with one or more embodiments ofthe present invention.

FIG. 10 is a flow diagram illustrating exemplary steps in the operationof an exemplary content recognition system, in accordance with one ormore embodiments of the present invention.

FIG. 11 is a flow diagram illustrating exemplary steps in the operationof an exemplary media device, in accordance with one or more embodimentsof the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Certain embodiments of the present disclosure may be found in a methodand system for detection of user-initiated events utilizing automaticcontent recognition (ACR). Various embodiments of the present inventionmay enable the detection of user-initiated events utilizing a systemthat is operable to perform automatic content recognition and that mayutilize an abstraction layer. Various embodiments of the invention mayutilize ACR techniques described herein to detect user interaction witha media device and/or a unit (for example, a set-top box) associatedwith (or included within) the media device. In response to detectingcertain user interactions, the content recognition system may cause aninteractive event to be communicated to and be executed on the mediadevice. A viewer may interact with the interactive event, which mayaffect the operation of the media device and/or the unit, for example, aset-top box.

In various embodiments of the invention, a method may be executed in amedia device that is operable to play video content. The method maycomprise receiving user interaction with the media device, wherein theuser interaction causes the initiation of a feature that affects videocontent that plays on the media device. One or more interactive eventsmay be received from a content recognition system in response to theuser interaction. The one or more interactive events may be executed inorder to affect video content that is playing on the media device, andone or more overlay windows may be displayed. The method may comprisecommunicating to the content recognition system one or more videofingerprints that correspond to video content that has been affected bythe user interaction. The content recognition system may communicate theone or more interactive events to the media device in response todetecting a match with the one or more video fingerprints. The one ormore interactive events may be executed at a time that is synchronizedto the user interaction with the media device.

In an exemplary embodiment of the invention, the video content asaffected by the user interaction may determine the one or moreinteractive events that are communicated from the content recognitionsystem. For example, the user interaction may cause the initiation of afeature that is operable to fast forward through one or morecommercials. The one or more interactive events, when executed, maydisplay an incentive to watch the one or more commercials and maydisplay an option whether to continue fast forwarding. The incentive towatch the one or more commercials may be an overlay window that informsviewers of a scavenger hunt game, and the scavenger hunt game may beoperable to allow viewers to earn points by indicating items in the oneor more commercials. In another example, the user interaction may causethe initiation of a feature that is operable to pause a program, and theone or more interactive events, when executed, may display entertainmentor advertisements while the video content is paused. In another example,the user interaction may cause the initiation of a feature that isoperable to tune to an unauthorized or blocked channel or program, andthe one or more interactive events, when executed, may block the mediadevice from displaying an unauthorized program and may tune to analternative channel.

In an exemplary embodiment of the invention, the method may compriseexecuting the one or more interactive events to accept user responsedata from a user. The user response data affects video content thatplays on the media device. In another exemplary embodiment, userpreferences may determine the type of the one or more interactive eventsthat are communicated from the content recognition system.

Various embodiments of the invention may utilize a content recognitionsystem comprising one or more servers. The content recognition systemmay be operable to detect one or more matches between one or more videofingerprints corresponding to video content playing on a media deviceand one or more video fingerprints corresponding to video content, forexample, archived content, in the content recognition system. Thecontent recognition system may be operable to detect user interactionwith the media device by analyzing the one or more fingerprint matches.The user interaction may initiate a feature that affects video contentthat plays on the media device. In response to the user interaction, theone or more servers of the content recognition system may be operable tocommunicate one or more interactive events to the media device. The oneor more interactive events may be operable to execute on the mediadevice and may be operable to control video content that plays on themedia device. In some embodiments of the invention, the contentrecognition system comprises an event trigger module that is operable toindicate to the one or more servers the one or more interactive eventsthat should be communicated to the media device.

In some embodiments of the invention, the content recognition systemcomprises an event trigger module that is operable to indicate to theone or more servers, one or more types of video playback situationsrelated to video content playing on the media device that should triggeran interactive event. The types of video playback situations maycomprise one or more of the following: when the video content starts,stops, pauses, fast forwards, reverses and changes playback speed, etc.In some embodiments of the invention, the one or more interactive eventsmay be adapted to present an incentive to watch a commercial if a viewerinitiates a feature that is operable to fast forward the video content.In some embodiments, the one or more interactive events may be adaptedto entertain a viewer or present advertisements while the video contentis paused.

In some embodiments of the invention, the content recognition system maybe operable to implement parental controls by detecting user interactionwith the media device by detecting the initiation of a feature that isoperable to tune to an unauthorized channel or program. In this regard,the one or more interactive events may be adapted to block the mediadevice from displaying an unauthorized program, and may tune to analternative channel.

FIG. 1 is a high-level block diagram illustrating an exemplary systemfor detection of user-initiated events, in accordance with one or moreembodiments of the present invention. Referring to FIG. 1, there isshown an ACR system with abstraction layer. The system 100 may comprisean real-time event manager 110, one or more fingerprint servers 120, oneor more fingerprint matching systems 130, an application server 140, anevent trigger module 150, a network 160, and one or more end-userdevices 170. A live network/broadcast feed (shown in FIG. 1) may beprovided to the system 100, and the same, time delayed, broadcast feedmay be read/analyzed by end-user devices 170 (media devices) connectedto system 100 and by fingerprint servers 120 (and for example othermodules) included in system 100. Each of the end-user devices 170 maycomprise an ACR module 172. FIG. 1 also shows a service provider 180that provides service to the end-user devices 170. FIG. 1 also shows afingerprint database (DB) of pre-recorded content 152 and an interactivetimeline module 154.

The real-time event manager 110 may comprise suitable logic, circuitry,code, and/or interfaces that may be operable to manage real-time eventsbased on inputs provided by one or more sources. For example, thereal-time event manager 110 may be operable to manage real-time eventsprovided by an event trigger module 150 that may assign interactiveevent IDs to live network programming. The real-time event manager 110may also manage interactive events (interactive event IDs assigned toarchived network programming) provided by an event trigger module 150,which may be prepared by a component that analyzes and archives videosegments. A network time protocol (NTP) server may be utilized toprovide synchronization of multiple fingerprint servers utilizingmultiple fingerprint technologies. Moreover, the real-time event manager110 may be operable to trigger interactive events in legacy systemsand/or in web-based systems.

The real-time event manager 110 may be operable to receive one or moreinputs from an event trigger module 150 and generate, based on thoseinputs, the interactive event identifiers (IDs) that can be communicatedto the fingerprint servers where they can be associated with or assignedto the video fingerprints generated by the fingerprint servers 120. Thereal-time event manager 110 may be operable to communicate theinteractive event IDs to a television system (e.g., legacy system)and/or to a web system. The interactive event identifiers may beutilized in the television system and/or in the web system to triggerinteractive events. Moreover, the communication of the interactive eventidentifiers may be based on one or more of an EBIF, an HTTP livestreaming (HLS), a satellite network protocol, or some other protocol.

In an embodiment of the invention, the real-time event manager 110 maybe operable to generate one or more signals that provide instructions tothe fingerprint servers 120 to enable the identification of a networktelevision station based on the logo, symbol, sign, watermark, and/ortext that are typically utilized to represent the network televisionstation. The instructions may be related to information that is to begenerated and/or provided to the end-user devices 170 for networktelevision station identification. The instructions may indicate thetype of information that is to be provided to the end-user devices 170and/or when such information is to be provided. In some instances, aportion of the ACR system 100 other than the real-time event manager110, or in conjunction with the real-time event manager 110, maygenerate the signals for providing instructions to the fingerprintservers 120.

The fingerprint servers 120 may comprise suitable logic, circuitry,code, and/or interfaces that may be operable to handle fingerprintingand fingerprint communications to the fingerprint matching systems 130.Since each vendor or television manufacturer is likely to utilize itsown fingerprint technology, each of the fingerprint servers 120 may be adedicated server for each of the fingerprint technologies supported bythe system 100. In some embodiments of the invention, a portion of oneor more fingerprint servers 120 may be operable to perform videofingerprinting while a portion may be operable to perform audiofingerprinting. Fingerprint technologies from multiple vendors mayutilize different computations to perform fingerprinting of video and/oraudio frames. For example, each fingerprint technology may utilize aspecific set of algorithms, parameters, operations, and/or dataprocessing methods, for example.

In an embodiment of the invention, the fingerprint servers 120 may beoperable to receive one or more signals from the real-time event manager110 and/or from another portion of the ACR system 100 to enable theidentification of a network television station based on the logo,symbol, sign, watermark, and/or text that are typically utilized torepresent the network television station. U.S. application Ser. No.13/730,422, discloses an exemplary method and system for logoidentification utilizing ACR in accordance with an embodiment of theinvention. The instructions may be utilized to determine and/or providelocations to the end-user devices 170 to take fingerprints of the videocontent being displayed on a viewing screen. In some instances, at leasta portion of the fingerprinting locations may be provided by thereal-time event manager 110 and/or by another portion of the ACR system100 through the instructions received by the fingerprint servers 120. Inother instances, the fingerprinting locations may be determined by thefingerprint servers 120 based on locally and/or remotely storedinformation. Each fingerprinting location may comprise coordinates in avideo frame (e.g., x coordinates, y coordinates) that indicate aparticular region in the video frame to fingerprint.

The fingerprint servers 120 may provide the fingerprinting locations forcommunication to the end-user devices 170, for example, in the form offingerprint profiles. The fingerprint profiles may comprisefingerprinting locations and/or other information to be utilized by anend-user device for ACR fingerprinting. In some instances, thefingerprint profiles may be generated by the fingerprint servers 120 inresponse to the instructions received. In other instances, thefingerprint profiles comprising the fingerprinting locations may bereceived by the fingerprint servers from the real-time event manager 110and/or from another portion of the ACR system 100. The fingerprintprofile of a particular end-user device 170 may be updated based on anindication that additional and/or different locations may be neededduring fingerprinting to identify the network television station logo orsymbol being displayed on a viewing screen at the end-user device 170.The update may be generated by the corresponding fingerprint server andthen communicated to the end-user device 170 or may be received by thecorresponding RTFS from the real-time event manager 110 and/or fromanother portion of the ACR system 100 and then communicated to theend-user device 170.

The indication that a fingerprint profile update may be needed may bethe result of network operations that recognize that certain content isbeing broadcast by several network television stations concurrently(e.g., State of the Union address). In such instances, thefingerprinting locations being utilized may not analyze the region in avideo frame where the logo of the network television station isdisplayed. Thus, providing additional fingerprinting locations in thisregion may enable detection and identification of the logo and,consequently, of the network television station.

The indication that a fingerprint profile update may be needed may alsobe the result of feedback provided by an end-user device 170. Thefeedback may indicate, for example, that the content being displayed hasbeen identified but that the content may have originated in any one ofseveral sources and the particular source of the content has not beenidentified. In such instances, the fingerprinting locations beingutilized may not analyze the region in a video frame where the logo ofthe network television station is displayed. Thus, providing additionalfingerprinting locations in this region may enable detection andidentification of the logo and, consequently, of the source of thecontent.

In some instances, the fingerprint profile and/or the fingerprintprofile update received by an end-user device may comprise informationthat indicates to the end-user device that any additional fingerprintinglocations may be utilized automatically when the source (e.g., networktelevision station) of a particular content is not initially identified.

In one or more embodiments of the invention, the fingerprint servers 120may be operable to communicate fingerprint profiles and/or fingerprintprofile updates to the end-user devices 170 through the fingerprintmatching systems 130. Feedback and/or queries from the end-user devices170 may be received by the fingerprint servers 120 for processing. Thefingerprint servers 120 may in turn communicate informationcorresponding to the feedback and/or queries from the end-user devices170 to the real-time event manager 110 and/or to another portion of theACR system 100 for further processing.

The fingerprint matching systems 130 may comprise suitable logic,circuitry, code, and/or interfaces that may be operable for matchingfingerprints produced by the fingerprint servers 120 with fingerprintsproduced by the end-user devices 170. Each of the fingerprint matchingsystems 130 may correspond to a particular ACR or fingerprinttechnology. In this regard, each of the fingerprint matching systems 130may be supported by a third party such as a TV manufacturer, forexample. Additionally, one or more of the fingerprint matching systems130 may be a hosted service, for example, cloud computing, etc., whichmay be accessible via the Internet.

The fingerprint servers 120 may be operable to send fingerprints,interactive event IDs and other information to their correspondingfingerprint vendors (fingerprint matching systems 130) through one ormore networks (e.g., wireline networks, optical, hybrid fiber coaxial(HFC), wireless networks) and/or by utilizing one or more communicationprotocols. Communication between the fingerprint servers 120 and thefingerprint matching systems 130 may occur through one or more wirelessand/or wireline communication links. The communication links describedabove may support one or more communication protocols. Accordingly, thefingerprint servers 120 and the fingerprint matching systems 130 maycomprise suitable logic, circuitry, code, and/or interfaces to enablethe use of the appropriate communication protocols.

The fingerprint matching systems 130 may be operable to comparefingerprints produced by the end-user devices 170 with fingerprintsprovided by the fingerprint servers. When a match occurs, a fingerprintmatching system 130 may indicate that an interactive event is to takeplace in an end-user device 170. These interactive events may allow aviewer of an end-user device 170 to be presented with information on thescreen or display of an end-user device (such as advertisements, promptsto change channels or playback modes, notifications of coupons, gamesand the like) and may allow a viewer to interact with the informationpresented.

Communications between the fingerprint matching systems 130 and theend-user devices 170 may occur through one or more communicationlinks/networks (e.g., wireline networks, optical, hybrid fiber coaxial(HFC), wireless networks), for example network 160, and/or by utilizingone or more communication protocols. The communication links and/ornetworks described above may support one or more communicationprotocols. Accordingly, the fingerprint matching systems 130 and theend-user devices 170 may comprise suitable logic, circuitry, code,and/or interfaces to enable the use of the appropriate communicationprotocols.

The application server 140 may comprise suitable logic, circuitry, code,and/or interfaces that may be operable to store or host (or containaddress information related to the location of) one or more applicationsor interactive events. It should be understood that a system (forexample, system 100) may include more than one application server, eventhough some of the figures and descriptions throughout this disclosuremay refer to a single application server in order to describe certainaspects of the systems and methods of this disclosure. It should beunderstood that in some setups, an end-user device may have anapplication installed prior to the time that a fingerprint match isdetected related to that device. In this setup, the application servermay push an interactive event that displays and/or executes on thepre-installed application. In other setups and/or situations, theapplication itself may be pushed to the end-user device and the end-userdevice may be operable to handle and/or execute the application andinteractive event. Various configurations may exist with respect to theend-user devices and the application server such that portions of thecode, which may be operable to execute an interactive event on anend-user device may be pre-installed on the end-user device. Therefore,throughout this disclosure, when reference is made to an application oran interactive event being pushed or communicated from an applicationserver, it may be understood that the application, or one or moreportions thereof, may be pre-installed on the end-user device.Alternatively, the application server may push the application includingthe interactive event together.

The application server 140 may be operable to receive from an end-userdevice 170 information related to an interactive event ID. Theinformation may be provided by the end-user device 170 in response to amatch between a fingerprint that may be taken and/or generated by theend-user device and a fingerprint that may be taken and/or generated bya corresponding fingerprint server 120. In some embodiments of thepresent disclosure, the application server 140 may receive informationrelated to an interactive event ID directly from a fingerprint matchingsystem 130 instead of or in addition to receiving information from anend-user device 170. Thus, in this regard, the end-user device mayprompt the application server 140 for content, or the application server140 may push the content to the end-user device without being promptedby the end-user device. Therefore, throughout this disclosure, wheneverreference is made to the end-user device requesting content from anapplication server (i.e., a pull), the application server may actuallypush the content to the end-user device, and vice versa with respect toreferences to application servers pushing content (i.e., the end-userdevice may instead request content from the application servers).

The event trigger module 150 may comprise servers, computers, devices,user interfaces and the like that allow interaction between anadministrative user (such as a partner or an event creator, director,etc.) and the system 100. Alternative, or in conjunction, the eventtrigger module 150 may allow for interaction with a component thatarchives and analyzes video segments. In one example, the event triggermodule 150 may include an end-user computer and peripheral devicesand/or interfaces such that an individual (such as a partner or an eventcreator, director, etc.) may communicate with the system 100 via thecomputer and peripherals and/or interfaces. The event trigger module 150may include peripherals and/or interfaces whereby a partner that isresponsible for creating event content that appears on end-user devicesmay create interactive events that synchronize to a live or replayedbroadcast. In another embodiment of the invention, the event triggermodule may include (or be in communication with) a component thatarchives and analyzes video segments such that interactive events may beassigned to archived video segments. For example, a component mayanalyze and store an episode of a TV show and may determine when in theepisode (or related commercial) an interactive event is to take place(for example, when to push a commercial to a media device). The eventtrigger module 150 may then communicate with an interactive timelinearchive to associate event IDs, which can then be communicated to areal-time event manager 110 to assign those interactive event IDs tofingerprints associated with live or replayed programming.

In operation, once the event trigger module 150 communicates with thereal-time event manager 110 to assign interactive event IDs tofingerprints associated with live or replayed programming being viewedon an end-user device 170, the real-time event manager 110 may generateand/or handle event identifiers or event triggers that correspond tospecific times in a program. The event identifiers may be assigned tothe appropriate fingerprints generated by the fingerprint servers 120.The real-time event manager 110 may provide synchronization toappropriately assign the event identifiers to the right spot on a videoand/or audio sequence.

Each of the fingerprint servers 120 may then communicate the eventidentifiers (generated or handled by real-time event manager 110) andthe fingerprints to its corresponding one of the fingerprint matchingsystems 130. The fingerprint matching systems 130 in turn receivefingerprints from their corresponding end-user devices 170 and try tomatch those fingerprints to the ones received from their correspondingfingerprint servers 120. When a match occurs, the event identifierand/or other information may be passed to the appropriate end-userdevice (and/or to an application server 140). Based on informationdetermined by the fingerprint matching system 130, the end-user devicemay obtain, for example, interactive information (e.g., graphics, text,applications) from the application server 140. There may be otheractions performed by the user in connection with the interactive eventand/or other information presented or produced in connection with theinteractive event.

Once a match occurs in one of the fingerprint matching systems 130 andthe end-user device 170 (and/or the application server 140) obtains theappropriate information from its corresponding fingerprint vendor, theapplication server 140 then sends the appropriate content (such as anapplication or interactive event) to the end-user device thatcorresponds to the interactive event ID. The end-user device maycommunicate interactive event ID information to the application server140 in order to request content or content may be pushed by theapplication server 140 to an end-user device without the devicerequesting it. In some embodiments, in order for an end-user device toaccept content from an application server 140, the device may have to belogged in to an appropriate network-connected application, web page orthe like. Additionally, the end-user device may need to have anappropriate ACR service running on the device.

The end-user devices 170 may comprise a plurality of devices such asconnected TVs, connected TV with paired handheld and/or mobile devices(also referred to as second-screen devices) such as smartphones, PDAsand tablets, for example. End-user devices may also be referred to asmedia devices, where an end-user device may be a media device that isutilized by a user to communicate with a system, such as an ACR system.The End-user devices may also be referred to as viewer devices, displaydevices, or ACR-based devices, for example. In a particular end-userdevice setup, for example a setup in a viewer's home, one or moredevices may be connected to a system 100. Each device in a setup may beconnected to the system via a network 160, or via a separate end-userdevice that is part of the same setup. For example, a viewer may have asetup that includes a smart TV that connects to the system 100 via thenetwork 160, and the viewer may also have a tablet (or othersecond-screen device) that communicates with the smart TV and allows theviewer to interact with content that is displayed on the smart TV. Insome instances, the content displayed on such a second-screen device maycoincide with the content that is displayed on the main end-user device,and this coincided content may allow a user or viewer to interact withcontent displayed on the main end-user device in a synchronized manner.In instances when a secondary or paired device that supports aparticular fingerprint technology is used, that device may also be ableto communicate with the corresponding fingerprint match (i.e., as in thecase of audio) system 120 that supports the compatible fingerprinttechnology. This may in effect comprise a manual pairing with presentedcontent on the primary screen (i.e., TV) and as such may operate in asimilar manner to pairing directly with the smart TV.

Each of the end-user devices 170 may comprise an ACR module 172. The ACRmodule 172 may comprise suitable logic, circuitry, interfaces and/orcode that may be operable to provide various functionality and/orservices comprising synchronous event manager functionality, fingerprintcapture services, logo detection, and/or user initiated event detection.

An end-user device 170 may include (or be in communication with) a mediaunit provided or supported by a television network or service provider,such as, for example, a set-top box, gateway, computer, tuner, otherdigital video device, etc. The media unit may read and/or analyze anetwork/broadcast feed (as shown in FIG. 1, for example) in order todisplay programming on the end-user device 170. The media unit mayinclude a digital video recorder (DVR) component that is operable torecord live programming for later replay or playback by a viewer,sometimes referred to as on demand viewing. An ACR system, for example,system 100, may be operable to analyze video being played back on anend-user device via a DVR the same as it analyzes live programming. Inthis respect, an ACR system may analyze video being played back via aDVR, generate fingerprints of the video, and send the fingerprints to afingerprint matching system (for example, systems 130). A media unit anda smart TV (end-user device) may also be configured and operate suchthat the smart TV may communicate commands to the media unit. Forexample, a smart TV (with proper authentication, for example) may beoperable to instruct a media unit to change channels, perform videoplayback functions, perform guide functions and the like. A media unitmay comprise suitable logic, circuitry, code, and/or interfaces, and mayutilize appropriate communication protocols to interact directly with anetwork, for example, the network 160. In this regard, a media unit maysend data via a network or may receive commands via a network.

End-user devices, for example utilizing an ACR service, API's, anapplication framework, and other interface, circuitry, software and thelike, may be operable to present applications to a viewer. For example,an end-user device may be operable to present text (including customizedformatting, including new lines, pages, fonts and the like), windowborders, wrappings and other graphics. Additionally, an end-user devicemay be operable to present customized overlay windows, for exampledesigned by third parties. An end-user device may be adapted to presentbrand graphics and messages, sponsorship text and logos, graphs andother feedback images, for example, user response results from a poll.

The end-user devices 170 may be operable to receive and utilize afingerprint profile and/or a fingerprint profile update and to takefingerprints in a pre-determined number of locations in a video frame.Each fingerprinting location may be defined by a set of coordinates thatdescribe a region in the video frame where a fingerprint of the videocontent is to be taken. The end-user devices 170 may be operable toreceive a series of fingerprint profiles and/or fingerprint profileupdates and may be operable to adjust ACR fingerprinting accordingly.

The abstraction layer may refer to, for example, the ability of an ACRsystem to assign the same event identifiers to different sets of videofingerprints that are generated by different video fingerprinttechnologies. That is, by appropriately timing the assignment of eventidentifiers to multiple sequences of video fingerprints that aregenerated from the same video content but with different videofingerprinting technologies, a single ACR system is able to supportvideo fingerprinting technologies from multiple vendors. This approachallows the ACR system to be both flexible and scalable with respect tofingerprint technology vendors. Additional details of an exemplary ACRsystem that utilizes an abstraction layer may be found in U.S. patentapplication Ser. No. 13/730,352, filed on Dec. 28, 2012, which is herebyincorporated herein by reference in its entirety.

FIG. 2 is a high-level block diagram illustrating a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention. Referring to FIG.2, there is shown a portion 200 of an exemplary system for detection ofuser-initiated events, which may comprise one or more end-user devices(devices 270 a, . . . , 270 c) and a network 260. The network 260 may bea network similar to the network 160 described in association withFIG. 1. As shown in FIG. 2, various end-user devices (media devices)such as connected (smart) TVs, smart TVs with paired hand-held(second-screen) devices, and even hand-held devices only may beconnected via a network 260 to an ACR system (for example, similar tothe system 100 of FIG. 1).

In one setup, a single smart TV (device 270 a) may be connected to anACR system. In another setup, multiple devices 270 b may be connected toan ACR system, for example where one device, such as a smart TV may bethe primary end-user device, and one or more hand-held devices, such astablets, PDAs or smart phones, may be second-screen devices. Asecond-screen device associated with the connected TVs may be a tablet(e.g., iPad, Samsung Galaxy, etc.), smartphone (e.g., iPhone, Android,etc.), or other like device, for example. In this setup, the secondscreen devices may either be in communication with the network 260directly, or they may be in communication with the network via theprimary device, or both. It should be understood that in any particularsetup, not all of the devices need to be ACR-enabled devices. In someexamples, one of the devices may an ACR-enabled device and the other maynot be ACR-capable, but may still interact with the ACR-enabled device.Therefore, throughout this disclosure, whenever reference is made to anACR-based device in a setup, it should be understood that not all of thedevices need to be ACR-capable. Devices in a setup may also be referredto as media devices. Furthermore, throughout this disclosure, wheneverreference is made to a single end-user device in a setup, it should beunderstood that the setup may contain multiple end-user devices, andvice versa with respect to references to multiple end-user devices. Inanother setup, hand-held devices 270 c, which may in some situationsoperate as second-screen devices, may act as one or more primaryend-user devices and optionally one or more secondary end-user devices.In this setup, one or more hand-held devices may receive broadcast feedcontent and may also allow one or more users to interact withinteractive events sent over the network 260.

An end-user device (for example, devices 270 a, . . . , 270 c) mayinclude (or be in communication with) a media unit (for example, set-topboxes 275 a, . . . , 275 c). The media unit may have been provided by orbe supported by a television network or service provider. The media unitmay read and/or analyze a network/broadcast feed (as shown in FIG. 2) inorder to display programming on the end-user device. In many respects,the media unit of FIG. 3 may be similar to the media unit described inconjunction with FIG. 1, for example. Viewers may interact with themedia unit via an associated remote control, or optionally via ahand-held device. For example, a viewer may interact with the media unitto start, stop, fast forward, reverse and/or pause content that isplaying on the end-user device. The viewer may also interact with themedia unit to select (or tune in to) a channel to watch, or to changechannels. The viewer may also interact with the media unit to view,manage, select and/or watch and/or replay recorded programming, and inthis regard, the viewer may interact with the media unit to start, stop,fast forward, reverse and/or pause the recorded content that is playingon the end-user device. Media units such as set-top boxes may comprisesuitable logic, circuitry, code, and/or interfaces that may be operableto present viewers with many more options and perform many morefunctions and/or operations, some of which may affect how the connectedend-user device operates. It should be understood that this disclosurecontemplates various types of media units (for example, set-top boxes,gateways, computers, tuners, other digital video devices, etc.)—not justset-top boxes. Therefore, throughout this disclosure, whenever referenceis made to a set-top box, it should be understood that other types ofmedia units are contemplated as well.

A set-top box (for example, unit 275 a) and a smart TV (end-userdevice), such as end-user device 270 a, may also be configured andoperable such that the smart TV may communicate commands to the set-topbox. For example, a smart TV (with proper authentication, for example)may be operable to instruct a set-top box to change channels or videoplayback modes, perform guide functions and the like. Alternatively orin addition, a set-top box may comprise suitable logic, circuitry, code,and/or interfaces, and may utilize appropriate communication protocolsto interact directly with a network (for example network 260). In thisregard, a set-top box may send data via a network or may receivecommands via a network. For example, a remote system, for example, anACR system, may send commands via a network to a smart TV and, providedthat the ACR system possesses proper authentication, may cause the smartTV to change channels, either directly on the TV itself or indirectlyvia an API exposing an interface between the TV and STB (e.g., IRfunction), or video playback modes, perform guide functions or otheroperations.

FIG. 3 is a high-level block diagram that illustrates a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention. Referring to FIG.3, there is shown a portion 300 of an exemplary system for detection ofuser-initiated events, which may comprise one or more end-user devices(devices 370 a, . . . , 370 d), one or more fingerprint matching systems(systems 330 a, . . . , 330 d), one or more fingerprint servers 320 a, .. . , 320 d, and an real-time event manager 310. As shown in FIG. 3,each end-user device (devices 370 a, . . . , 370 d) may support and/orutilize a different fingerprint technology, and therefore each end-userdevice may require its own fingerprint matching system (systems 330 a, .. . , 330 d). Accordingly, an ACR system may be operable to includefingerprint servers capable of communicating with some or all of thedifferent fingerprint matching systems supported by the system. FIG. 3,shows how fingerprint servers 320 a, . . . , 320 d may be operable tocommunicate with distinct fingerprint matching systems 330 a, . . . ,330 d. These features may be implemented by having a central videofingerprint repository and management platform that facilitatestriggering interactive events across various platforms regardless of theACR vendor solution (e.g., fingerprinting technology). Such a system maybe scalable and additional ACR vendor solutions may be added. Anreal-time event manager 310 may be operable to assign one or moreinteractive event identifiers to the different sets of videofingerprints (for example utilizing different fingerprint technologies)generated by the fingerprint servers 320 a, . . . , 320 d.

The End-user devices (for example, devices 330 a, . . . , 330 d) mayutilize the same video and/or audio fingerprinting technology utilizedby the associated fingerprint servers and supported by the associatedfingerprint matching system vendors. The fingerprint vendor may be avendor, manufacturer, or other third-party service provider that maycomprise suitable logic, circuitry, code, and/or interfaces operable toprovide hosted services over the Internet for the viewer devices. Theseservices may comprise video and/or audio fingerprint matching as well asthe delivery of any interactive event IDs associated with a match. Theend-user devices may comprise suitable logic, circuitry, code, and/orinterfaces that may be operable to take and send fingerprints to thefingerprint vendor for matching.

FIG. 4 is a high-level block diagram that illustrates a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention. Referring to FIG.4, there is shown a portion 400 of an exemplary system for detection ofuser-initiated events, which may comprise one or more end-user devices(devices 470 a, . . . , 470 d), one or more application servers 440, oneor more system API's 442 (Application Programming Interface) and/or oneor more vendor-specific interfaces 444 a, . . . , 444 d.

As shown in FIG. 4, a variety of end-user devices 470 a, . . . , 470 dmay be operable to properly execute applications served by applicationserver 440 by way of, for example, an API such as the system API 442and/or vendor-specific interfaces 444 a, . . . , 444 d. A standardizedAPI may be utilized to reduce the number of APIs being utilized. In oneexample, a vendor (for example, vendor 1) may acquire the system API 442and then implement the system API 442 (along with additional vendor1—specific interfacing) into the devices manufactured by the vendor. Inthis example, the system API 442 and the interface for vendor 1 444 amay be implemented fully or substantially within devices manufactured byvendor 1. In another example, a vendor (for example, vendor 1) mayprovide a vendor-specific interface or API (such as interface 444 a) tothe operators and/or directors of application server 440 (or otherparties that are responsible for creating applications and/orinteractive events). In this example, operators, directors or otherparties may implement an API for each vendor, in which case the vendorinterface 444 a and system API 442 may be implemented within theapplication server 440, or related system components utilized to createand store applications.

API's that allow interfacing with various systems and sub-systems suchas systems or modules from different vendors may be provided. Forexample, an API may enable an end-user device to perform real timedownloading and invocation of applications from the application server440. Additionally, API's and/or vendor interfaces may establish a singleframework that can be leveraged for various applications. For example,the framework may include functions that facilitate user response datato multiple locations, for example, user response data could be sent tomultiple distinct locations and/or servers, the address of which may bemaintained by the system or network content provider. In anotherexample, the framework may enable the invocation of overlay windows andapplications, and may facilitate data communication between multipleapplications and/or overlay windows, and between applications and anexternal entity, for example a third-party service. In another example,the framework may enable the communication of content, data and otherinformation to and from second-screen devices.

End-user devices, for example, devices 470 a, . . . , 470 d, maycomprise an ACR service that enables the device to perform one or morefunctions that may allow the end-user device to operate with orcommunicate with the ACR system and may allow the end-user device toexecute one or more applications and/or interactive events. In someexamples, the ACR service may be active and running directly onconnected devices whenever the devices are operating or communicatingwith the system, and the ACR service may invoke an applicationframework, which may include system API's. The ACR service may beoperable to “listen” for or detect actionable events (for example,interactive events) and then act upon these events according to rules,which may be defined in a corresponding interface definition. The ACRservice may be operable to receive and/or collect user responses inresponse to an interactive event, for example, and communicatecorresponding response data to an application, service or externalentity such as a server, service, URL, or the like.

FIG. 5 is a high-level block diagram that shows a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention. Referring to FIG.5, there is shown a portion 500 of an exemplary system for detection ofuser-initiated events, which may comprise one or more end-user devices(for example, device 570), an interactive director 550, one or moreapplication servers 540 and a network 560. FIG. 5 also illustrates theflow of information that may occur when an event trigger module 550initiates an interactive event related to live or replayed programmingand causes (in connection with an NTP server, a fingerprint server and afingerprint matching system—not shown in FIG. 5) that event to bedisplayed on network-connected end-user devices.

In the example shown in FIG. 5, the event trigger module 550 mayindicate to the application server 540, which applications and/orinteractive events the application server 540 should push tonetwork-connected end-user devices when certain content (and/or contentplayback situations/indications) appears on the screen of the end-userdevice 570. The event trigger module 550 may select appropriateapplications and/or interactive events that are synchronized with timepoints in the live or replayed content being played on the end-userdevice 570. In this regard, the event trigger module may communicateinteractive events that correspond to points in live or replayedcontent, such that the real-time event manager 110 may be operable totrigger interactive events, via a fingerprint server and fingerprintmatching system, in the end-user device 570. When an interactive eventis triggered in an end-user device, for example, device 570, the devicemay send a request for data via a network 560 to an application server540. Note that the triggered events may have specific parametersdefining their operation during trick modes (i.e., pause, fast-forward,rewind, slow, etc).

The application server(s) 540 may be operable to handle data sent froman end user device 570, as well as data sent from various components andmodules within the event trigger module 550. When the application server540 receives a signal and/or request, and an interactive event ID froman end-user device 570, the application server 540 may send applicationdata 542 to one or more end-user devices 570 via a network 560. Theapplication server 540 may send application data 542 to an end-userdevice 570 in response to receiving a signal (and an interactive eventID) from the device 570 for example. The application server 540 and/orthe event trigger module 550, may also be operable to integrate with avariety of back-end systems, for example to integrate externalapplication/event/content servers (not shown) into the system.

It should be understood that application data (for example, applicationdata 542) relates to one or more interactive events and/or applications.Therefore, when a description herein refers to a module communicating anapplication or an interactive event, it should be understood that theapplication and/or the interactive event may actually be sent in theform of one or more data packets. One or more data packets related to anapplication and/or interactive event may generally be referred to asapplication data, for example application data 542. The application data542 may comprise, but need not be limited to, text, banners, shapes,graphics, overlays, sidebars, applications, widgets, and/or video, forexample. The application data 542 may also comprise computer code,applications or other interactive elements that when executed by anend-user device, may present content to a viewer and allow a viewer tointeract with an interactive element. The application data 542 may besynchronized with content that is currently (or was recently) displayedon the end-user device. In one example, when the application data 542 issent to an end-user device 570, the end-user device may display anoverlay window 572 that takes up at least a portion of the full display574.

In one example implementation of the techniques described herein, forexample as shown in FIG. 5, a viewer may be watching a replay of atelevision show, using a DVR feature in a set-top box, for example,where the video of the television show is displaying on the screen ofthe end-user device 570. In this example, the television show that theviewer is watching may have previously been analyzed and archived by acomponent associated with (or included in) the event trigger module 550.The component may have determined when in the show an interactive eventshould take place, or what types of video playback situations shouldtrigger a particular interactive event (e.g., trick mode options).

An ACR system, for example, one similar to the system 100 described inreference to FIG. 1, may be operable to detect when a user initiates anevent related to video that plays on an end-user device. For example, anACR system may be operable to detect when a user starts, stops, fastforwards, reverses and/or pauses a program that is playing on theend-user device. An ACR system may also be operable to detect when auser selects, or tunes to a channel, or changes channels. An ACR systemmay detect these and other user-initiated events by utilizing the ACRtechniques described herein. In addition to being operable to detectwhen video displayed on an end-user device matches one or more referencevideo segments, an ACR system may also be operable to detect when videodisplayed on an end-user device is playing back in a different mannerthan one or more reference video segments. For example, the ACR systemmay detect that a video is being played back at different speed or in adifferent direction, for example. An ACR system may also be operable todetect when video displayed on an end-user device is playing back in adifferent manner than one or more reference video segments by detectingimages (that display on the screen of an end-user device) related tocontent playback commands. For example, if a box and/or icons appear onthe end-user device when a user chooses to fast forward a recordedprogram, an ACR system may detect the box and/or icons to determine thata viewer is attempting to fast forward a program. For example, theviewer may initiate a feature that is operable to fast forward throughone or more commercials. In addition to or in conjunction with ACRtechniques, an ACR system may detect user-initiated events by acceptingand analyzing signals sent via the end-user device (and/or a relatedunit such as a set-top box) over a network.

Referring to the example illustrated in FIG. 5, a viewer may be watchinga replay of a television show where the video of the television show isdisplaying on the screen of the end-user device 570. The event triggermodule 550 (and/or associated components) may have determined what typesof video playback situations should trigger a particular interactiveevent. For example, the event trigger module 550 may have determinedthat if during (or just before) the first commercial in a recordedprogram the user attempts to fast forward, an interactive event shouldbe initiated. The screen 574 illustrates an exemplary video frame thatis related to commercial that a viewer may encounter and attempt to fastforward past. In some embodiments of the invention, the ACR system maydetect that the video displaying on the screen of the end-user device570 is playing back at a greater speed than an archived reference video(which indicates that the viewer is attempting to fast forward). The ACRsystem (including a fingerprint server and a fingerprint matchingsystem) may then initiate an interactive event (with an associatedinteractive event ID) to execute on the end-user device 570. Theend-user device may then request the interactive event and/or theapplication that may be associated with the interactive event ID fromthe application server 540. The application server 540 may then send theappropriate application data 542 to the end-user device 570 via anetwork 560. The device may then display the interactiveevent/application.

FIG. 5 also shows an exemplary interactive event that may display on anend-user device in response to a viewer attempting to fast forwardthrough a commercial. The interactive event, when executed by theend-user device, may display an overlay window 572 that is synchronizedwith the user-initiated event (for example, attempted fast forwarding).The interactive event may be designed, for example, to persuade orincentivize a viewer to continue to watch a commercial. The interactiveevent may inform the user of some reason why they may want to continuewatching the commercial and then may solicit a viewer response and/orchoice. In one example, the interactive event may solicit user responsevia a multiple choice question, although other types of responses arecontemplated, such as text responses. Once a viewer sees the interactiveelement on the screen of an end-user device, the user may interact, forexample by choosing one of the choices in a multiple choice question.The selection of a particular choice may be made utilizing a remotecontrol and/or a second-screen device in communication with the end-userdevice, for example.

The interactive event (displayed via an overlay window 572, for example)may be designed, for example, to persuade or incentivize a viewer tocontinue to watch a commercial by informing the viewer of usefulinformation or by allowing the viewer to interact with a contest,program, feature or the like. In one example, as shown in FIG. 5, ascavenger hunt contest may allow a viewer to collect points by viewingand identifying items that appear on the screen of an end-user deviceduring a commercial. When the viewer attempts to fast forward acommercial, the ACR system may initiate an interactive event thatdisplays an overlay window 572 informing the viewer that they can earnscavenger hunt points or other types of incentives for this commercial.The viewer may then have the option to continue fast forwarding orinteract while fast forwarding, as this will be controlled by the STBwhile the ACR module may be resident on the smart TV itself, or to stopfast forwarding and watch the commercial. The playback of the program(including any fast forwarding) may be paused until the user selects andoption. This may happen if the ACR enabled smart TV can communicate withthe STB and the viewer has permitted this access. A scavenger huntcontest like the one contemplated by FIG. 5 may be associated with anonline service that allows a user to collect points across multipleprograms and optionally to compare the number of points earned withother members of the online service. The user's points and otherinformation related to the scavenger hunt contest may be associated witha user account on a system that supports a plurality of user accounts.The end-user device may be adapted to communicate via a network withsuch an online service to store point totals and retrieve informationabout other members.

Other types of contests, programs, features or the like may be displayedto a viewer instead of, or in addition to, a scavenger hunt contest. Forexample, overlay window 572 may inform a viewer that the commercialcontains information about valuable coupons, or optionally that thecommercials include trailers for exciting new movies.

The end-user device, associated set-top box, and/or the ACR system maybe adapted to allow viewers to set preferences related to interactiveevents and related to content that is displayed on the end-user devices.For example, a viewer may maintain a list of movies (or genres) that maybe of interest to the viewer. User preferences may indicate thatinteractive events should only notify the viewer of movie trailers thatare categorized accordingly. In another example, user preferences mayindicate that no overlay windows associated with the fast forwardfeature should be displayed. In another example, user preferences mayindicate that no overlay windows should be displayed at all. Theend-user device, associated set-top box, and/or the ACR system mayinclude a mode or setting whereby in one or more situations, userpreferences may be overridden. If such a mode or setting is active, anoverlay window may be presented on an end-user device even if a usersetting indicates an opposite preference.

FIG. 6 is a high-level block diagram that shows a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention. Referring to FIG.6, there is shown a portion 600 of an exemplary system for detection ofuser-initiated events, which may comprise one or more end-user devices,for example devices 670 a and 670 b. FIG. 6 also illustrates the flow ofinformation that may occur when a viewer responds to an interactiveevent that is displayed on the screen of an end-user device. Part or allof one or more responses sent by an end-user device in response to aninteractive event may be referred to as user response data.

The example depicted in FIG. 6 includes a two-device (devices 670 a, 670b) setup. In this example, once a viewer sees an overlay window 672 onthe screen of their end-user device (device 670 a and/or 670 b), theviewer may interact with the related application or interactive elementvia a second-screen device 670 b, such as a PDA, tablet or smart phone.In this example, the user may have an application installed on thesecond screen device 670 b that enables the user to see displays,information and interactive choices that coincide with the overlaywindow 672 on the screen of the end-user device 670 a. The viewer maythen select a choice, or otherwise indicate a response by interactingwith the second-screen device 670 b, for example by pressing on thescreen with a stylus or the viewer's finger, or by pressing buttons onthe second-screen device.

As one example, depicted in FIG. 6, it can be seen that the screen ofsecond-screen device 670 b may display interactive choices that coincidewith the choices displayed in overlay window 672. A viewer may thenindicate a response using the second-screen device 670 b, for example,by selecting a button labeled “No.” In the example of FIG. 6, when theviewer selects “No” on the second-screen device, the same choice appearsas selected in the overlay window 672. The second-screen device 670 bmay communicate the viewer's response to the end-user device 670 a ordirectly to upstream application servers by a variety of connectionssuch as a wired connection and/or a wireless connection (e.g., WIFI,Bluetooth, infrared, and the like). Other setups may include a singleend-user device, for example device 670 a. In these setups, a viewer mayinteract with the related application or interactive element via aremote control or other interface associated with the end-user device670 a.

Referring to FIG. 6 for example purposes, a viewer may indicate aresponse that he does not want to fast forward through a commercial (seethe “No” option indicated on overlay window 672). For example, theviewer may be interested in participating in a scavenger hunt contest,and may choose to watch the commercials in order to look for items thatdisplay in the commercials to gain points. In this example, if theviewer selects “No,” the commercials may resume playing back on theend-user device 670 a. This may occur if the ACR enabled smart TVcommunicates directly with the STB/DVR. In most scenarios, this may notbe the case. The viewer will explicitly have to stop fast forwarding ifthey wish to participate in the game.

FIG. 7 is a high-level block diagram that shows a portion of anexemplary system for detection of user-initiated events, in accordancewith one or more embodiments of the present invention. Referring to FIG.7, there is shown a portion 700 of an exemplary system for detection ofuser-initiated events, which may comprise one or more end-user devices(for example devices 770 a and 770 b) and a network 760 in connectionwith an ACR system and/or other systems, servers and the like. FIG. 7also illustrates the flow of information that may occur when a viewerresponds to an interactive event that is displayed on the screen of anend-user device, for example to participate in a scavenger hunt contest.Part or all of one or more responses sent by an end-user device inresponse to an interactive event may be referred to as user responsedata.

In the example depicted in FIG. 7, a viewer may have selected “No” inresponse to an overlay window that may have been displayed or presentedwhen the viewer attempted to fast forward a commercial. When the viewerselected the “No” option, which may have indicated that the viewer wasnot sure he wanted to fast forward, the commercials may have resumedplaying back on the end-user device 770 a, although in most cases theviewer will have to explicitly stop fast forwarding. In this example,the viewer may have decided to watch the commercials in order toparticipate in a scavenger hunt contest where a user, for example,notices and indicates items displayed on the end-user device. In theexample depicted in FIG. 7, a user may have noticed an item, such as apower outlet, which may have appeared in the background of a commercialdisplayed on the screen 774 of an end-user device 770 a. The user mayindicate an item that was noticed using a hand-held device, for example,device 770 b, or a remote control or other interface associated with enduser device 770 a. For example, the display of a hand-held device 770 bmay coincide with the content that is displayed on the screen or couldpermit drawing and/or highlighting of that content on the end-userdevice 770 a. The hand-held device may be operable to allow a user totouch or circle noticed items using a finger or a pointer, stylus or thelike. If a user discovers and indicates an item that qualifies forpoints in a scavenger hunt contest, an overlay window 772 may display onthe screen informing the viewer that one or more points were earned.

If a viewer responds to an interactive event that requires data to besent to a remote system, user response data 742 may be transmitted via anetwork 760 to such a remote system, for example an ACR system or othersystem, server or the like. The remote system, for example an ACRsystem, may be operable to receive and handle user response data. Theuser response data 742 may comprise, but need not be limited to, answersto multiple choice questions, text responses, contest point updates andthe like. In one example, user response data 742 may be sent to the sameapplication server (similar to application server 540 of see FIG. 5)that transmitted the application data and/or interactive event to theend-user device 770 a. However, in some embodiments of the invention,the end-user device may be operable to transmit response data to otherservers and devices in addition to or instead of an application server.

The exemplary implementation of the techniques described herein as shownand described in relation to FIGS. 5-7 is only one example. The ACRtechniques described herein may be applied to other situations, and maybe utilized to initiate other interactive events.

In one situation, a viewer may be watching a television show (eitherlive or replayed) where the video of the television show is displayingon the screen of an end-user device. Referring to FIG. 5 for explanationpurposes only, the event trigger module 550 may have determined that ifa show is paused (utilizing DVR features, for example incorporated intoa set-top box), an interactive event should be initiated. For example,the ACR system may detect that the video that is displayed or presentedon the screen of the end-user device 570 is playing back at a slowerspeed (or paused completely) than an archived reference video (whichindicates that the viewer has paused the show). The ACR system, thefingerprint matching system and associated components, may then initiatean interactive event (with an associated interactive event ID) toexecute on the end-user device 570. The invocation of the event may beperformed locally on the device as the detection of the trick mode(i.e., pause, fast forward, rewind) will in most cases be performedthere as well. The end-user device may then request the interactiveevent and/or application associated with the interactive event ID fromthe application server 540. The application server 540 may then send theappropriate application data 542 to the end-user device 570 via anetwork 560. The device may then display the interactiveevent/application.

The interactive event, when executed by the end-user device, may displayan overlay window 572 that may be synchronized with the user-initiatedevent, for example, a pause a show. The overlay window may take up aportion of the screen or the whole screen 574. The interactive event maybe designed to entertain a viewer while the show is paused. For example,the interactive event may be a game, quiz, or other type ofentertainment. The interactive event and end-user device(s) may beadapted to accept user response data from a viewer and optionallycommunicate user response data to a remote system.

In another example, while a show is paused, the ACR system may initiatea commercial and/or advertisement to display on the screen of theend-user device 570. Such commercials and/or advertisements may or maynot be interactive. In some embodiments of the invention, the ACR systemmaintains an archive of time-relevant commercials that may be pushed toend-user devices at certain times or during certain playback situationsduring a program. In some situations, commercials may be relevant andvaluable to a network for a limited amount of time, for example, 7 days.In some situations, a television network or other content provider mayonly get credit and/or money for commercials that are played toconsumers within a certain number of days. In this regard, it may bebeneficial for a content provider to utilize an ACR system and adatabase to maintain a set of time relevant commercials. An ACR systemmay detect when end-user devices are paused, for example, and may pushrelevant commercials to end-user devices. In other examples, an ACRsystem may detect when a user may be watching an older recorded program,for example, when the commercials begin to play, and the ACR system maypush time relevant commercials to replace stale or less valuablecommercials.

In another situation, the ACR techniques described herein may be appliedto implement parental controls. For example, a child may be watching atelevision show, either live or replayed, where the video of thetelevision show is displaying on the screen of an end-user device.Referring to FIG. 5, the event trigger module 550 may have determinedthat if a viewer attempts to tune to a specific unauthorized channel orprogram, an interactive event should be initiated. For example, a viewermay initiate a feature that is operable to tune to channels, and thecontent recognition system may be operable to detect the channel thatthe viewer is attempting to tune to. Unauthorized channels or programsmay be, for example, pornographic channels, violent channels and otherchannels that are not appropriate for children or may have been blockedfor some other reason. For example, the ACR system may detect that thevideo of the new channel or program displaying on the screen of theend-user device 570 matches archived reference video that has beenflagged as an unauthorized channel or program or it may use a variant oflogo and/or guide detection to determine the channel number and compareagainst a list of prohibited channels for a given service provider.Alternative or in addition, the ACR system may detect other featuresdisplayed on the screen to detect an unauthorized channel or program,such as a channel and/or program information bar along the bottom of thescreen. Alternative or in addition, a set-top box may transmitinformation to the ACR to system to indicate the channel/program asbeing unauthorized. The ACR system and optionally a fingerprint matchingsystem, may then initiate an interactive event, with an associatedinteractive event ID, to execute on the end-user device 570. Theend-user device may then request the interactive event and/orapplication associated with the interactive event ID from theapplication server 540. The application server 540 may then send theappropriate application data 542 to the end-user device 570 via anetwork 560. The device may then display the interactive event and/orapplication.

The interactive event, when executed by the end-user device, may displayan overlay window 572 that is synchronized with the user-initiated event(for example, tuning to an unauthorized channel/program). The overlaywindow may be overlaid or displayed on at least a portion of the screen574. The interactive event may be designed to block the video and/oraudio of the unauthorized program and/or prevent the viewer from tuningto the unauthorized channel. For example, the interactive event maydisplay an overlay window 572 that informs the viewer that they areattempting to access an unauthorized channel and/or program and mayoptionally provide the viewer with alternative options. For example, ifthe overlay window may provide the viewer with an option to tune to acartoon channel instead. In some embodiments of the invention, theinteractive event may cause the end-user device and/or set-top box totune to an alternative channel automatically. Alternatively, or inconjunction with presenting a viewer with alternative channel options,an interactive event may display advertisements, for exampleadvertisements targeting children. In one example, the interactive eventmay be operable to display advertisements that take up the majority (orall) of the screen such that the content of the unauthorizedchannel/program cannot be seen by the viewer.

FIG. 8 is a flow diagram 800 that illustrates exemplary steps in theoperation of an exemplary system for detection of user-initiated events,in accordance with one or more embodiments of the present invention. Atstep 802, a program and/or video is playing on a media device that is incommunication with an ACR system. At step 804, a user interacts with themedia device and/or caused initiation of an event. For example, the usermay start to fast forward through a commercial, pause the program orattempt to tune to an unauthorized channel and/or program. At step 806,the ACR system may detect a video playback situation. For example, theACR system may detect that the user started to fast forward through acommercial, paused the program or attempted to tune to an unauthorizedchannel and/or program. At step 808, the ACR system may communicate aninteractive event to the media device in response to the video playbacksituation. At step 810, the media device may execute the interactiveevent, and the interactive event may cause an overlay window to bepresented or displayed on the media device. For example, the overlaywindow may present an incentive for the user to watch a commercial, orit may present a game that the user can play while the program ispaused, or it may present alternative channels that the user can tune toinstead of an unauthorized channel.

At step 812, the user may interact with the interactive event and/or maymake a choice related to the interactive event and/or video playback.For example, a user may choose whether to watch the commercial or resumefast forwarding, or a user may play a game that displayed while aprogram was paused, or a user may choose to tune to an alternate channelinstead of an unauthorized channel. At step 814, the interactive eventmay execute to affect video content on the media device, based on theuser interaction. For example, a commercial may be played that offers aviewer chances to earn points in a scavenger hunt game, or a game may bepresented until a user chooses to resume video playback, or the mediadevice may tune to an alternate channel chosen by the user. At step 816,in some situations, the interactive event may receive and/or handle userresponse data, and in some situations, the media device may communicateuser response data to a remote system and/or server. For example, if auser chooses to watch commercials to earn points in a scavenger huntgame, the media device may communicate user account information and gameinformation to a remote system.

FIG. 9 is a block diagram that shows an exemplary data processing systemthat may be used in accordance with one or more embodiments of thepresent invention. Additionally, the systems, methods and/or techniquesdescribed herein may utilize more than one data processing system. Forexample, referring to FIG. 1, the fingerprint servers 120 may berealized in one or more data processing systems. The real-time eventmanager 110 may be realized in one or more data processing systems. Theevent trigger module 150 may be realized in one or more data processingsystems. The application server 140 may be realized in one or more dataprocessing systems. The fingerprint matching systems 130 may be realizedin one or more data processing systems. The end-user devices may includeone or more data processing systems. Additionally, one or more of thecomponents of system 100 shown in FIG. 1 may be realized within a singledata processing system. One of skill in the art will realize thatdifferent combinations of data processing systems that realizecomponents of system 100 may be provided without departing from thespirit and scope of the invention.

Referring to FIG. 9, there is shown a data processing system 900 thatmay correspond to one or more components of an exemplary system fordetection of user-initiated events. The data processing system 900 maycomprise a processor module 910, a memory module 920, a network module930, and an input/output (I/O) interface module 940.

The processor module 910 may comprise suitable logic, circuitry, code,and/or interfaces that may be operable to perform the operations,functions, processes, techniques, and the like, for example one or moreof the operations, functions, processes and techniques described herein.In this regard, the processor module 910 may be operable to enable oneor more components of a system for detection of user-initiated events.The processor module 910 may comprise at least one processing device912. The processing device 912 may be a central processing unit (CPU), adigital signal processor (DSP), and/or other type of integrated circuitthat may be utilized to perform data processing operations.

The memory module 920 may comprise suitable logic, circuitry, code,and/or interfaces that may be operable to store information utilized toenable an example dial testing and audience response system. The networkmodule 930 may comprise suitable logic, circuitry, code, and/orinterfaces that may be operable to allow the data processing system 900to communicate with one or more other data processing systems. This maycomprise, for example other data processing systems within the samesystem for detection of user-initiated events, or other data processingsystems external to the same system for detection of user-initiatedevents. The network module 930 may be operable to support one or morecommunication protocols such as wireline protocols and/or wirelessprotocols. The I/O interface module 940 may comprise suitable logic,circuitry, code, and/or interfaces that may be operable to interact withone of more devices or other sources of interactive information such asinterfaces or peripherals that allow interaction by a director and/oradministrator, or a device that allows a computer code to be loaded intothe memory module 920 of the data processing system.

In various embodiments of the invention, and referring to FIG. 10, amethod 1000 may be executed in a content recognition system comprisingone or more servers that are operable to communicate with one or moremedia devices. At step 1004, the content recognition system may enabledetecting one or more matches between one or more video fingerprintscorresponding to video content playing a media device and one or morevideo fingerprints corresponding to video content in the contentrecognition system. At step 1008, the content recognition system mayenable detecting user interaction with the media device by analyzing theone or more fingerprint matches. The user interaction may initiate afeature that affects video content that plays on the media device. Atstep 1010, the one or more servers may communicate one or moreinteractive events to the media device. The one or more interactiveevents may be operable to execute on the media device and may beoperable to control video content that plays on the media device. Theone or more interactive events may be synchronized to the userinteraction with the media device.

In some embodiments of the invention, as shown at step 1002 as apreliminary step, an event trigger module in the content recognitionsystem may indicate to the one or more servers one or more types ofvideo playback situations related to video content that should triggeran interactive event. The event trigger module may also indicate one ormore interactive events that may be communicated to the media device asa result of a video playback situation being detected. The types ofvideo playback situations may comprise one or more of the following:when the video content starts, stops, pauses, fast forwards, reversesand changes playback speed, etc. In one example, the one or moreinteractive events may be operable to present an incentive to watch acommercial if a viewer initiates a feature that is operable to fastforward the video content. As another example, the one or moreinteractive events may be adapted to entertain a viewer or presentadvertisements while the video content is paused. In some embodiments,the content recognition system is operable to implement parentalcontrols, and the content recognition system may detect the initiationof a feature that is operable to tune to an unauthorized channel orprogram. In these embodiments, the one or more interactive events may beadapted to block the media device from displaying an unauthorizedprogram, and may tune to an alternative channel.

In some embodiments of the invention, as shown at step 1006 as anintermediary step, the content recognition system may be operable tomaintain and/or reference user preferences related to interactiveevents, and the content recognition system may reference userpreferences to determine types of interactive events that the contentrecognition system should communicate to the media device.

In various embodiments of the invention, and referring to FIG. 11, amethod 1100 may be executed in a media device that is operable to playvideo content. At step 1102, the media device may receive userinteraction from a viewer, wherein the user interaction initiates afeature that affects video content that plays on the media device. Inone example, the user interaction may comprise initiating a feature thatis operable to fast forward through one or more commercials, and the oneor more interactive events, when executed, may present an incentive towatch the one or more commercials and may display an option whether tocontinue fast forwarding. In another example, the user interaction maycomprise initiating a feature that is operable to pause a program, andthe one or more interactive events, when executed, may presententertainment or advertisements while the video content is paused. Inanother example, the user interaction may comprise initiating a featurethat is operable to tune to an unauthorized channel or program, and theone or more interactive events, when executed, may block the mediadevice from displaying an unauthorized program and tune to analternative channel.

At step 1104, the media device may generate one or more videofingerprints that correspond with video content that is playing on themedia device, as the video content is affected by the user interaction.At step 1106, the media device may receive one or more interactiveevents from a content recognition system in response to the userinteraction. At step 1108, the media device may execute the one or moreinteractive events to affect video content that is playing on the mediadevice and the media device may display one or more overlay windows. Thedisplaying may occur at time that is synchronized to the video contentplaying on the media device. At step 1110, the media device may acceptuser response data from a user, wherein the user response data affectsvideo content that plays on the media device. For example, a user mayindicate that they do not want to fast forward through a commercial, inwhich case the video content may resume playing back on the mediadevice.

In various embodiments of the invention, a method may be executed in amedia device 570 that is operable to play video content. The method maycomprise receiving user interaction 1102 with the media device, whereinthe user interaction causes the initiation of a feature that affectsvideo content that plays on the media device 570. The method maycomprise receiving 1106 one or more interactive events 542 from acontent recognition system 100 in response to the user interaction. Themethod may comprise executing 1108 the one or more interactive events toaffect video content that is playing on the media device 570, anddisplay one or more overlay windows 572. The method may comprisecommunicating to the content recognition system 100 one or more videofingerprints that correspond to video content that has been affected bythe user interaction. The content recognition system 100 may communicate1010 the one or more interactive events 542 to the media device 570 inresponse to detecting a match 1008 with the one or more videofingerprints. The one or more interactive events 542 may be executed ata time that is synchronized to the user interaction with the mediadevice.

In an exemplary embodiment of the invention, the video content asaffected by the user interaction may determine the one or moreinteractive events 542 that are communicated 1010 from the contentrecognition system 100. For example, the user interaction may cause theinitiation of a feature that is operable to fast forward through one ormore commercials, and the one or more interactive events 542, whenexecuted, may display an incentive 572 to watch the one or morecommercials and may display an option 572 whether to continue fastforwarding. The incentive 572 to watch the one or more commercials maybe an overlay window 572 that informs viewers of a scavenger hunt game,and the scavenger hunt game may be operable to allow viewers to earnpoints by indicating items in the one or more commercials. In anotherexample, the user interaction may cause the initiation of a feature thatis operable to pause a program, and the one or more interactive events542, when executed, may display entertainment or advertisements whilethe video content is paused. In another example, the user interactionmay cause the initiation of a feature that is operable to tune to anunauthorized or blocked channel or program, and the one or moreinteractive events 542, when executed, may block the media device fromdisplaying an unauthorized program and may tune to an alternativechannel.

In an exemplary embodiment of the invention, the method may compriseexecuting 1108 the one or more interactive events to accept 1110 userresponse data from a user, wherein the user response data affects videocontent that plays on the media device 570. In another exemplaryembodiment, user preferences may determine the type of the one or moreinteractive events that are communicated from the content recognitionsystem.

Another embodiment of the present disclosure may provide anon-transitory machine and/or computer readable storage and/or media,having stored thereon, a machine code and/or a computer program havingat least one code section executable by a machine, computer and/or dataprocessing system, thereby causing the machine, computer and/or dataprocessing system to perform the steps as described herein for detectionof user-initiated events utilizing automatic content recognition.

Accordingly, the present disclosure may be realized in hardware,software, or a combination of hardware and software. The presentdisclosure may be realized in a centralized fashion in at least onemachine, computer and/or data processing system; or in a distributedfashion where different elements are spread across severalinterconnected machines, computers and/or data processing systems. Anykind of machine, computer and/or data processing system or otherapparatus adapted for carrying out the methods described herein issuited. A typical combination of hardware and software may be ageneral-purpose computer system with a computer program that, when beingloaded and executed, controls the computer system such that it carriesout the methods and techniques described herein.

The present invention may also be embedded in a computer programproduct, which comprises all the features enabling the implementation ofthe methods described herein, and which when loaded in a computer systemis able to carry out these methods. Computer program in the presentcontext means any expression, in any language, code or notation, of aset of instructions intended to cause a system having an informationprocessing capability to perform a particular function either directlyor after either or both of the following: a) conversion to anotherlanguage, code or notation; b) reproduction in a different materialform.

While the present invention has been described with reference to certainembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted withoutdeparting from the scope of the present disclosure. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the present invention without departing from its scope.Therefore, it is intended that the present invention not be limited tothe particular embodiment disclosed, but that the present invention willinclude all embodiments falling within the scope of the appended claims.

What is claimed is:
 1. A method, comprising: in a media device that isoperable to play video content, wherein: the media device comprises anautomatic content recognition (ACR) module that communicates with an ACRsystem that is external to the media device, and the ACR system supportsdifferent video fingerprint technologies from multiple vendors, wherein:the ACR system assigns same interactive event identifiers to differentsets of video and audio fingerprints that are generated from the samevideo and audio content by the different video and audio fingerprinttechnologies from the multiple vendors based on timing of the assignmentof the interactive event identifiers; and the interactive eventidentifiers are utilized by the media device to trigger one or moreinteractive events; the ACR module providing fingerprints to afingerprint matching system in the ACR system for matching by thefingerprint matching system; the ACR module receiving information and/orcontent from the ACR system based on the matching by the fingerprintmatching system; and the media device: detecting by the ACR module, userinteraction with the media device, wherein the user interaction causesthe initiation of a feature that affects video content that plays on themedia device; receiving by the ACR module, the one or more interactiveevents from the fingerprint matching system in the ACR system inresponse to the user interaction; and executing the one or moreinteractive events to: affect video content that is playing on the mediadevice; display one or more overlay windows corresponding to theaffected video content; and allow interaction via the displayed one ormore overlay windows.
 2. The method according to claim 1, comprisingcommunicating to the fingerprint matching system in the ACR system oneor more video fingerprints that correspond to video content that hasbeen affected by the user interaction, wherein the fingerprint matchingsystem in the ACR system communicates the one or more interactive eventsto the media device in response to detecting a match with the one ormore video fingerprints.
 3. The method according to claim 1, wherein:the ACR system comprises an event trigger module that provides anindication of the one or more interactive events to be communicated tothe media device; and the one or more interactive events are executed ata time that is synchronized to the user interaction with the mediadevice.
 4. The method according to claim 1, wherein the video content asaffected by the user interaction determines the one or more interactiveevents that are communicated from the ACR system.
 5. The methodaccording to claim 4, wherein: the user interaction causes theinitiation of a feature that is operable to fast forward through one ormore commercials; and the one or more interactive events, when executed,display an incentive to watch the one or more commercials and display anoption whether to continue fast forwarding.
 6. The method according toclaim 5, wherein: the incentive to watch the one or more commercials isan overlay window that informs viewers of a scavenger hunt game; and thescavenger hunt game is operable to allow viewers to earn points byindicating items in the one or more commercials.
 7. The method accordingto claim 4, wherein: the user interaction causes the initiation of afeature that is operable to pause a program; and the one or moreinteractive events, when executed, display entertainment oradvertisements while the video content is paused.
 8. The methodaccording to claim 4, wherein: the user interaction causes theinitiation of a feature that is operable to tune to an unauthorized orblocked channel or program; and the one or more interactive events, whenexecuted, block the media device from displaying an unauthorized programand tune to an alternative channel.
 9. The method according to claim 1,wherein: executing the one or more interactive events comprisesaccepting user response data from a user; and the user response dataaffects video content that plays on the media device.
 10. The methodaccording to claim 1, wherein user preferences determine the type of theone or more interactive events that are communicated from the ACRsystem.
 11. An system, comprising: a media device that is operable toplay video content, wherein the media device comprises an automaticcontent recognition (ACR) module that communicates with an ACR systemthat is external to the media device, and the ACR system supportsdifferent video fingerprint technologies from multiple vendors, wherein:the ACR system assigns same interactive event identifiers to differentsets of video and audio fingerprints that are generated from the samevideo and audio content by the different video and audio fingerprinttechnologies from the multiple vendors based on timing of the assignmentof the interactive event identifiers; and the interactive eventidentifiers are utilized by the media device to trigger one or moreinteractive events; the ACR module providing fingerprints to afingerprint matching system in the ACR system for matching by thefingerprint matching system; and the ACR module receiving informationand/or content from the ACR system based on the matching by thefingerprint matching system; and the media device being operable to:detect by the ACR module, user interaction with the media device,wherein the user interaction initiates a feature that affects videocontent that plays on the media device; receive by the ACR module, theone or more interactive events from the fingerprint matching system inthe ACR system in response to the user interaction; and execute the oneor more interactive events to: affect video content that is playing onthe media device; display one or more overlay windows corresponding tothe affected video content; and allow interaction via the displayed oneor more overlay windows.
 12. The system according to claim 11, whereinthe media device is operable to communicate to the fingerprint matchingsystem in the ACR system one or more video fingerprints that correspondto video content that has been affected by the user interaction, whereinthe fingerprint matching system in the ACR system communicates the oneor more interactive events to the media device in response to detectinga match with the one or more video fingerprints.
 13. The systemaccording to claim 11, wherein: the ACR system comprises an eventtrigger module that provides an indication of the one or moreinteractive events to be communicated to the media device; and the mediadevice is operable to execute the one or more interactive events at atime that is synchronized to the user interaction with the media device.14. The system according to claim 11, wherein the video content asaffected by the user interaction determines the one or more interactiveevents that are communicated from the ACR system.
 15. The methodaccording to claim 14, wherein: the feature that affects video contentthat plays on the media device is operable to fast forward through oneor more commercials; and the media device is operable to execute the oneor more interactive events to display an incentive to watch the one ormore commercials and display an option whether to continue fastforwarding.
 16. The method according to claim 15, wherein: the incentiveto watch the one or more commercials is an overlay window that informsviewers of a scavenger hunt game; and the scavenger hunt game isoperable to allow viewers to earn points by indicating items in the oneor more commercials.
 17. The method according to claim 14, wherein: thefeature that affects video content that plays on the media device isoperable to pause a program; and the media device is operable to executethe one or more interactive events to display entertainment oradvertisements while the video content is paused.
 18. The methodaccording to claim 14, wherein: the feature that affects video contentthat plays on the media device is operable to tune to an unauthorized orblocked channel or program, and the media device is operable to executethe one or more interactive events to block the media device fromdisplaying an unauthorized program and tune to an alternative channel.19. The method according to claim 11, wherein: the media device isoperable to execute the one or more interactive events to accept userresponse data from a user; and the user response data affects videocontent that plays on the media device.
 20. The method according toclaim 11, wherein user preferences determine the type of the one or moreinteractive events that are communicated from the ACR system.
 21. Asystem, comprising: a media device; an automatic content recognition(ACR) module within the media device; an ACR system comprising an eventtrigger module; and a fingerprint matching system in the ACR system,wherein: the ACR System supports different video fingerprinttechnologies from multiple vendors; the ACR system assigns sameinteractive event identifiers to different sets of video and audiofingerprints that are generated from the same video and audio content bythe different video and audio fingerprint technologies from the multiplevendors based on timing of the assignment of the interactive eventidentifiers; the interactive event identifiers are utilized by the mediadevice to trigger one or more interactive events; and the ACR modulebeing operable to: communicate fingerprints to the fingerprint matchingsystem in the ACR system, wherein the fingerprint matching systemmatches the communicated fingerprints; and receive information and/orcontent from the ACR system based on the matching that is done by thefingerprint matching system; the media device being operable to: detectuser interaction with the media device that causes the initiation of afeature that affects video content that plays on the media device;receive the one or more interactive events from the fingerprint matchingsystem in the ACR system in response to the user interaction, whereinthe event trigger module provides an indication of the one or moreinteractive events that are to be communicated to the media device; andexecute the one or more interactive events to:  affect video contentthat is playing on the media device;  cause display one or more overlaywindows corresponding to the affected video content; and  allowinteraction on the media device via the displayed one or more overlaywindows.